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Abstract 

Necessary and sufficient conditions for approximation of a general channel by a general source are proved. For 
the special case in which the channel input is deterministic, which corresponds to source simulation, we prove a 
stronger necessary condition. As the approximation criteria, vanishing variational distance between the original and 
the approximated quantity is used for both of the problems. Both necessary and sufficient conditions for the two 
£NJ ' problems are based on some individual properties of the sources and the channel and are relatively easy to evaluate. 

In particular, unlike prior results for this problem, our results do not require solving an optimization problem to test 
^ ' simulatability. The results are illustrated with several non-ergodic examples. 

I. Introduction 
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The problem of simulating a random process is of fundamental importance in many research disciplines, such 
as speech processing, applied mathematics, computer science, and the design and performance evaluation of 



CZ2 

q , communication systems. Within information theory, one of the motivations is to understand a folklore result which 
says that 'the encoder output of an optimal source code with vanishing error probability is almost uniformly 
random.' Han [1] proves that the claim is indeed true when being almost uniformly random is defined as the 
vanishing normalized divergence distance between the output and the uniform distribution. Gray [2] proves the 
claim for rate-distortion codes and J-distance, which is defined by Ornstein [3]. In the literature, there are explicit 

00 ■ 

q ■ random number generator constructions based on source codes, of which basic motivation is the aforementioned 
^ \ folklore (cf. [4], [5], [6] and the references therein). 

The first work on simulation problem dates back at least 40 years, to von Neumann's algorithm to create 
independent fair coin flips from biased coin flips [7] and Elias' [8] improvement and fundamental limits. Work 
within computer science aims to develop 'efficient algorithms' (in terms of computational complexity) to simulate 
random processes (cf. [9], [10], [11] and references therein). Work within information theory has focused on 
determining the fundamental limits of the problem in the large-delay limit. In this paper, we follow the information 
theoretic approach and prove necessary and sufficient conditions for the simulation problem in the following two 
different setups. These necessary and sufficient conditions are similar to others in information theory in that, while 
they are not identical, they are similar in essence. 

The first problem we consider is determining whether a general coin source^ can approximate another general 
target source^ By approximation, we mean there should exist a deterministic mapping from realizations of one 

The authors are with the School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, 14853, USA. E-mail: 
ya68@ Cornell, edu, wagner@ece. Cornell, edu. 

'By general source, we mean a collection of random variables with no consistency requirements among them. 

2 Here, we adopt the convention used in [12], where target (resp. coin) source refers to the to be approximated (resp. to be used for 
approximation) random source. 
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source to the realizations of the other source, such that the resulting source 'well-approximates' the original one 
in some precise sense. This problem is also known as the probability distribution approximation problem. 

The second problem is finding necessary and sufficient conditions for determining whether a general coin source 
can approximate a general channel, given that another fixed general source is the input to the channel. Similarly, 
approximation means finding a deterministic mapping from the realizations of the input source and the coin source 
to the output alphabet of the channel, such that the joint distribution of the input and the simulated output is close 
to the true distribution. Observe that the channel simulation problem subsumes the source simulation problem, since 
the latter is a special case of the former with a deterministic input source. 

For the special case of the source simulation problem, for which both coin and target sources are stationary 
processes with finite states, [8] obtains a complete solution for this particular version of the problem. Followup 
works have proposed more efficient algorithms in terms of computational complexity (cf. [13], [14] and references 
therein) and\or universal algorithmic (cf- [15], [16] and references therein). 

The case, in which at least one of the sources (or the channel) is non-stationary and non-ergodic, has only received 
attention more recently. The fundamental work of Han and Verdu [17], introduces the information spectrum method, 
which is the standard tool to handle non-stationary and non-ergodic extensions of the problems in information theory. 
The basic problem, called approximation theory of output statistics, defined in their paper is closely related to the 
simulation problem, but different from the problems considered in this paper. A somewhat 'dual' of this problem, 
in which the coin source is a general source and the target source is i.i.d. fair coin flips is also solved. For this 
particular case, Vembu and Verdu [18] proved necessary and sufficient conditions. Furthermore, Steinberg and Verdu 
[19] considered the problem of simulating a general source using i.i.d. fair coin flips. They proved necessary and 
sufficient conditions for this problem in the aforementioned work. For the channel simulation problem, Steinberg 
and Verdu [20] proved necessary and sufficient conditions for the special case where the coin source is i.i.d. fair 
coin flips. All of these works incorporate fundamental notions from the information spectrum method, namely 
sup-inf entropy rates and sup-inf conditional entropy rates. 

The general case, where both the target random variable and the coin random variable are arbitrary sources, 
is also investigated in the literature. Results due to Nagaoka [21] (full proofs are also available in [12]) states 
a necessary condition and a sufficient condition in terms of sup and inf entropy rates of the target and the coin 
sources over countable alphabets, however there is a sizable gap between these two conditions, in other words the 
result is not conclusive. Nagaoka and Miyake [22] state necessary and sufficient conditions without such a gap, for 
the case of finite alphabet sources. However, these conditions are stated in terms of an optimization problem over 
all joint distributions with the marginals equal to target and coin sources' distribution, hence hard to evaluate. 

Our contributions in this paper may be summarized as follows: 

• We state new necessary and sufficient conditions, which are essentially the same, for a coin source to be an 
approximating source of a given target source, where both of the sources are general sources over countable 
3 By universal algorithm, we mean an algorithm which does not rely on the statistics of the coin source. 
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alphabets. Our necessary condition is strictly stronger than its state-of-the-art counterpart stated in [22]. 

• We state the first, to the best of our knowledge, necessary and sufficient conditions, which are essentially the 
same, for a coin source to be an approximating source of a channel, given a fixed input source to the channel, 
where both coin and input sources and the channel are general ones over countable alphabets. 

• Unlike the existing conditions in the literature, our conditions do not include a 1/n scaling factor. 

Note that both necessary and sufficient conditions for both of the problems are in terms of the intrinsic properties 
of the sources and the channel, in which the whole spectrum is exploited, as opposed to the traditional quantities 
like entropy and conditional entropy rates, which are the limiting points of the entropy and conditional entropy 
spectrum, respectively. Hence, this kind of approach may lead to solutions to some of the open problems of the 
non-stationary and non-ergodic information theory, such as [23]. 

The paper is organized as follows. In Section [TT] we state our notation used throughout the paper, give basic 
definitions and state our results. We also include examples to illustrate our results. Section [III] is devoted to the 
proof of sufficient and necessary conditions for source simulation problem and the demonstration of the fact that our 
necessary condition is strictly stronger than its state-of-the-art counterpart, while Section [IV] consists of the proof 
of sufficient and necessary conditions for channel simulation problem. The paper ends with conclusions, stated in 
Section El 



Boldface letters denote vectors; regular letters with subscripts denote individual elements of vectors. Furthermore, 
capital letters represent random variables and lowercase letters denote individual realizations of the corresponding 
random variable. Throughout the paper, all logarithms are base-e, unless otherwise specified. For p G [0, 1], H(p) 
denotes the binary entropy function. 1 = 1U {— oo,+oo} denotes extended real numbers. (R,B,fj,) denotes a 
measure space, with B denoting the Borel-sigma algebra on real numbers and /x denoting the Lebesgue measure. 
For an arbitrary sequence of real-valued random variables {Z n }^ ( L 1 , p — liminf n _+ 00 Z n = {a : lim^^oo Pr{Z n > 
a} = 0} denotes the "limit infimum in probability", (cf. Definition 1.3.1. of [12]). Q ~ f7[0, 1] is a shorthand 
notation for "U is a uniform random variable over [0,1]". Given a random variable X with p.m.f. Px, Ep x [-] 
denotes expectation with respect to Px- 

B. Definitions and Statement of the Results 

Definition 2.1: Given two random variables X, Y G X, such that X is countable set, with pmfs Px and Py, 
respectively; the variational distance between Px and Py, denoted by d(Px, Py), is defined: 



Note that we will also use d(X, Y) to denote the variational distance throughout the rest of the paper, interchangeably 
with d(Px,PY) to denote the quantity in (Q]). 



II. Notation, Definitions and Statement of the Results, Examples 



A. Notation 




(1) 
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Definition 2.2: Let X = {Xn\^ =1 and Y = {l" n }^Li ^ e two g enera l sources, where for all n G Z + , X n and Y n 
are random variables taking values in X n and y n , respectively, such that X n and y n are countable sets. We say that 
X is an approximating source for Y, if there exists a sequence of deterministic mappings^ {4> n ■ Xn — > 3^n}^i, 
such that lim n ^oo d (Y n , (j) n (X n )) = 0. 

Definition 2.3: Let X = {X,,}^ and Z = {^ n }^=i be arbitrary general sources, where for all n G Z + , X n 
and Z n are random variables taking values in X n and Z n , respectively, and X n and Z n are countable sets with 
a given couplingj Pz n \x n between them. Let, ~W Y \x = {Wy„|x„(^n|^n)}°^_ 1 be a general channel, where for 
all n G Z + , Wy n \x n denotes a conditional pmf over y n x Af n , where y n is countable set. We say that Z jj an 
approximating source for ~Wy\x> given X, if there exists a sequence of mappings {ip n : X n x Z n —>■ y n }%Li* such 
that lim n ^oo d (X n Y n , X n cp n (X n , Z n )) = 0. 

We state our necessary and sufficient conditions in terms of the following quantity. 

Definition 2.4: Given a random variable Z n taking values in Z n , where Z n is countable set, let S n {Z) denote 
an ordered list of z G Z n sequences with non-zero probability, from highest probable to lowest probable, i.e. 

S n (Z) = s.t. PzSzi) >■■■> PzM) >■■■■ (2) 

Next, using S n {Z), define the following partition of [0, 1] 

A = {0 = 5 <£i<...<1}, (3) 

such that Vi G Z + , 5, — 5i-i = Pz n {/k)- F° r anv 8 £ [0, 1), 

c *(«5)^log— i— , (4) 
Pz n (Zk) 

such that 5 G [^-1, (5^) for some k G Z + by using ©. Note c^(5) is a well-defined quantity. 

Observe that, c^(<5) > 0, for all n and for all 5 G [0, 1). Further, c^{5) is non-decreasing and right continuous in 

S. 

Remark 2.1: Throughout the rest of the paper, when we refer to a quantity including \i (which stands for Lebesgue 
measure), it should be explicitly understood that we are using (R, B, fi) as our measure space. 
Next, we state our main results: 

1) Source Approximation: Consider X = {X n }^ =l and Y = {Y n }^ =l , where for all n G Z + , X n is a random 
variable taking values in X n , such that X n is countable set (resp. Y n is a random variable taking values in 
y n , such that y n is countable set). 
Sufficient Condition : If 

V 7 G E, lim n (5 G [0, 1) : c x n {8) - <%{8) < 7) = 0, (5) 

n—roo 

4 A11 of the mappings, which are mentioned throughout the rest of the paper, are deterministic ones and for the sake of convenience, we 
drop the quantifier 'deterministic' from now on. 

5 For any pair of random variable such a coupling exists, i.e. product distribution of marginals gives a joint distribution of X n and Y n . In 
fact, as far as practical application goes, this case is the most interesting case. 
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Fig. 1. Graphical representation of c z (-) for Z £ {zi, Z2, z$, Z4, z 5 } with the distribution Pz(zi) = 0.025, Pz{z2) = 0.075, Pz(zz) = 0.2, 
Pz(z 4 ) = 0.3 and P z {z 5 ) = 0.4. 



then X is an approximating source for Y. Note that, (|5]) can also be written as 

/z-Uminf{c£(<5) - c y n (5)} = 00, (6) 

n^oo 

where fi — lim inf is the analogous of p — lim inf quantitiy using Lebesgue mesaure instead of the probability 
measure. 

Necessary Condition: If X is an approximating source for Y, then 

inf lim inf inf {<(«5 + e) - > 0. 

0<e<l n-*oo 0<<5<l-e 

2) Channel Approximation: Consider X = {J„}™ =1 and Z = {Z n }™ =1 , where for all n, X n is a random 
variable taking values in X n , such that X n is countable set (resp. Z n is a random variable taking values in Z n , 
such that Z n is countable set) with a coupling Pz n \x n - Further, ~W Y \x = {^y„|x„0'nl-^n)} OO _ 1 > where for 
all n G Z + , Wy n \x n denotes a conditional pmf over y n x X n , with being countable set. For all n G Z + , 
for any x G <Y n and 5 G [0, 1), dn X (5, x) (resp. c™(<5, x)) denotes the quantity defined in Definition 12.41 for 

Pz n \xM x ) ( res P- Wy.lXn (-In- 
sufficient Condition: If 



V 7 G R, lim E Pav L(S G [0, 1) : c z J x (5,X n ) - c™(5,X n ) < 7) 



0. 
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then Z is an approximating source for W, given X. 

Necessary Condition: If Z an approximating source for W, given X, then 

Ve G (0, 1), V 7 G R+ lim E P]Cn L(d G [0, 1 - e) : + e,X n ) - c™(5,X n ) < - 7 ) 



0. 



Remark 2.2: For the sake of comparison, now we state the necessary and sufficient conditions stated in [22]. 
Let X n (resp. y n ) be a set with |A? n | < oo (resp. \y n \ < oo) for all n G Z + and X n (resp. Y^) be a random 
variable defined over X n (resp. J^n) with distribution Px n (resp. iV n )- Define X = {X n }^ =1 and Y = {Y n }^ =1 
and V(X n ,Y n ) = {{P Xn ,Y„( x n, Y n )}™ =1 : The marginal distributions are P x „ and Py n , Vra G Z+}. 

. If 

sup p — lim inf < — log — — log — — — — - > > 0, (7) 



{P Xn , Yn (X n ,Y n )}z> =1 eV(X„,Y„) n -*°° i n p x n (X n ) n P Yn (Y n 
then X is an approximating source for Y. 
• If X is an approximating source for Y, then 

sup p - lim inf { -log — ) - - log tttt } > 0- (8) 

We demonstrate in Section IIII-CI (cf. Remark 13.2b that our necessary condition is strictly stronger than the one 
given in (8). Moreover, our conditions do not involve an optimization problem, and hence are easier to evaluate, 
as the following examples show. 

C. Examples 

In this section we provide the following examples to illustrate the necessary and sufficient conditions for source 
and channel simulation. 

Example 1: Let {X™} n >i, {X2}n>i, {Y™} n >i and {Y2}n>i be stationary, memoryless Bernoulli sources with 
parameters p\, p2, q\ and q2, respectively. Observe that we have [24] 

Vi G {1, 2}, - log — — -» H(pi) (a.s.) as n -> oo, (9) 
n P Xz [X- ) 

Vi G {1, 2}, - log - 1 -> (a.s.) as n ^ oo. (10) 

n fy,(i ] 

Next, we define coin and target sources as mixtures of these in the following way: 

VnGZ+, X n = Q 1 X? + (l-Q 1 )X 2 t , (11) 
VnGZ+, Y n = Q 2 Y? + (1 - Q 2 )Y2\ (12) 

where Q\ (resp. Q2) denotes an independent Bernoulli random variable with parameter a G [0,1/2] (resp. (3 G 
[0, 1/2]). Let X = {X n } n>1 (resp. Y = {Y n } n>1 ) denotes the coin (resp. target) source. Suppose Vn G Z+, Qi X 
X? X X£ and Q 2 X X YJ. Further, suppose that we have 

< qi < pi < q 2 < P2 < 1/2. (13) 



Using ©, COB, (HB and CE) we have 

Urn |-^2 - j = F(pi)l {0 <i<a} + ^(P2)l {a<5 <i } - #(gi)l { o< 5</3 } - tf(g 2 )l {/3<5 <i } , (14) 

for any 5 G [0, 1), such that 5 ^ a and 5 ^ j3. 

Next, we consider the following cases for a and /?. 
1) a < f3: Using ([13]) and ([14]) we have 

V<5g[0,1), &.L8^a,6^P lim - > min{H0 2 ) - iT(g 2 ), #(pi) - Hfa)} > 0. (15) 



n n 



(IT5T > immediately implies that 

3 7 G M + , s.t. lim n ( 5 G [0, 1) : ^ - ^ < = 0, 

n— >oo y n n J 

which implies that 

V 7 G R+, lim n(S G [0, 1) : <£(<$) - <*(<*) < 7) = 0. (16) 

n— >oo 

Recalling the sufficient condition for source approximation, ([ToT l implies that X is an approximating source 
for Y. 

2) a > [3 : Define M+9e = and consider any 5 G f + f J. Recalling (O and CI]), this implies 

]im i<<Mj)_m\ =HM . H{q2)<0 , 



n n 



which immediately implies that 



inf liminf inf - C M. }< . (17, 

eG(0,l) n-»oo S£[0,l-e) [ U TL 



inf liminf inf {ctib + e) - c y J5)\ < 0. (18) 

ee(0,l) n^oo Se[0,l-e) 



Using ([TT] ). we conclude that 



Recalling the necessary condition for source approximation, (1181) implies that X is not an approximating 
source for Y. 

Example 2: Let {X"} n >i, {Xjjn^i, {Z™} n >i and {Zgln^i be stationary, memoryless Bernoulli sources with 
parameters n, r 2 , pi and p2» respectively. Define 

VnGZ+, Z n = QiZ? + (l-Qi)Zj, (19) 
VnG Z+ X n = Q 2 X? + (1 - Q 2 )X2, (20) 

where Q\ (resp. Q 2 ) is Bernoulli random variable with parameter a G [0,1/2] (resp. 9 G [0,1/2]). Let X = 

{X n }~ =1 (resp. Z = {Z n }^° =1 ) denotes the input (resp. coin) source. Suppose Vn G Z+ Qi, Q 2 , XJ, XJ, Z?, Z£ 

are independent. Further, we define our channel in the following way 

( W^ n]Xn , with probability /?, 
y " |x " ~ 1 W^ n|x „, with probability 1 - p, { ' 
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where for i G {1,2}, W y „| X „ (y n |x n ) = YYj=i ^yix^'l 3 ^')' an( * ^Y|x * s a ^SC w ^ crossover probability 
G (0, 1/2) and Wy| X = {Wy™|X"}^i- Equivalently, we can define Y n , output of the channel due to the input 
X n , in the following way. First, let {U™} n >i and {U2} n >i denote stationary, memoryless Bernoulli sources with 
parameters qi and q 2 , respectively. Then, for any n G Z + , define = X n © U™ and Y?? = X n © U?, ■ Hence, we 



have 



A 



+ (1 - Q 3 )Y%, 



(22) 



where Q3 is a Bernoulli random variable with parameter (3. Suppose Vn G Z + , Qi X Q2 X Q3 X X™ X XJ? X 
Z^IZ^IUJIU^. Further, suppose that we have 



< qi < pt < q 2 < p 2 < 1/2. 



(23) 



Observe that for any x n G {0, l} n , ^ log 



1 



,(Y»|x») 



^ log Pc/ „( U n ) , V« G {1, 2}. Using this along with (1221 . 



and recalling (0} (observe the independence of Z and X), we have 



lim 

n— >oo 



-^(Pl) 1 ^^} + #(P2)l{a<5<!} - -H" O?l)l{0<5</?} - -^fe)l{/3<5<l}, (24) 



n n 

for all {x 11 }™ =1 and for all 5 G [0, 1) except 5 = a and 5 = (3. 
Next, we consider the following cases for a and (5. 
1) a < f3: First of all, observe that using d23l) , we have 

3M G M + , s.t. M < min{H{p 2 ) - H(q 2 ),H( Pl ) - H( qi )}. 

Plugging (|25]> in ([24]) yields 



lim Ep x 

n— >oo 

immediately implies that 
sup < 7 G M : lim Ep 
which directly implies that 



V7 G M, lim Ep x „ 

n— »oo 



M [8 G [0, 1) : 



c z J x (5,X n ) c^&X* 



< M 



n 
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(25) 



(26) 



Ac[n . j_^X_Q <ff(£,X") 
A* G [0, 1) : = = < 7 



n 



n 



mUg[0,1) : (ff,X»)-^( < y,X n )< 7 



> > M, 



0. 



(27) 



Recalling the sufficient condition for channel approximation, (1271 ) implies that Z is an approximating source 
for Wy|x> given X. 



2) a > (3 : Let R + B e = 2=£. Using (O and (J23), we have 



lim 

n— >oo 



t& |x (<y + e,X n ) C(5,X* 



< -7, 



(28) 



a+/3 



for any 7 G (0, H(q 2 ) - H{ Pl )), 8 € ( (3, , 

A 4 ( ( > 



and x™. Since a > /3, we have 

a + /3 ->o. 



(29) 
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Combining (|28]> and (|29]> yields 

3e G (0, 1), 3 7 G R + , s.t. lim E Pxr , 



H \8 G [0, 1 - e) : 



c^(5 + e,X") c^X* 



n 



< -7 



> 0. 
(30) 



Recalling the necessary condition for channel approximation, (1301 implies that Z is not an approximating 
source for Wyix> given X. 

Note that since the channels we mix in Example 2 are memoryless binary symmetric channels, the resulting necessary 
and sufficient conditions are independent from the input source distribution, therefore the problem reduces to source 
approximation problem. Hence, Example 2 demonstrates (possibly in an exaggerated manner) the close relationship 
between source and channel approximation. 

Example 3: Let {Z™} n >i be stationary memoryless Bernoulli source with parameters pi, for i G {1,2}, such 
that < pi < p2 < 1/2. For any n G Z + , define 



Z n = QiZ? + (l-Qi)Z£, 



(31) 



where Qi is a Bernoulli random variable with parameter < a < 1/2. We denote the coin source as Z = {Z n }^ =l . 
Further, for any n G Z + , define 



X n = Q 2 X? + (1 - Q 2 )X 



2 - 



(32) 



where Q2 is a Bernoulli random variable with parameter < j3 < 1/2 and Px™ = TYj=i for i G {1,2} 

and 

1/2, if x G {0,1}, 
0, if x = 2, 



(33) 



and 



Px 2 (x) 



0, if x G {0,1}, 

1, if x = 2. 



(34) 



We denote input source as X = {X"}™ =1 . Moreover, for i G {1, 2} define the following channels 

f l- ft , if (x,y) = (0,0) or (x,y) = (1,1), 

<7i, if (x, y) = (0, 1) or (x, y) = (1, 0), 

1, if (x, y) = (2,2), 

0, else, 



(35) 



with < q\ < q2 < 1/2. Using 051 ), we define the following general channel. 



{Wy n \ Xn , with probability a, 
WL X „, with probability 1 - a, 



(36) 



A 



where for i G {1,2}, W^„ |x „ = n"=i ^yix^il^i) with W Y\x is as defined in <E3) and < a < 1/2. Let 
= {WV«|X"}nLi denote the general channel. 
Further, suppose that 

X?±Z?±Q h (37) 
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Fig. 2. Channel denned in d35b - 



Vn G Z + , Mi G {1, 2}. Observe that using ([37]), we have 

ct(5^ n ) = c z n {5), 
for any 5 G [0, 1) and x n G {0, 1, 2} n . Hence, (EU) yields 

Cn X (5 X n ) 

lim — = H(pi)l {0<s<a} + H(p 2 )l {a<s< i}, 



(38) 



(39) 



for any 5 G [0, 1) except 5 = a and for any x n G {0, 1, 2} n . Further, (recall ©) (|34]>, ([35]> and ([36]> implies that 

(40) 
(41) 



V5 G [0, l)\a, Vx" G {0, l} n , lim ' = #(<7i)l{o<5<a} + #fe)l {a<5 <i } ; 



V5 G [0, 1), Vx n G {2} n , lim ^ 1 = . 



Next, consider the following possibilities: 



1) H(p 2 ) > H(q 2 ), H{ Pl ) > H( qi ): Let M = mm{H(p 2 ) - H(q 2 ),H( Pl ) - H( gi )} > 0. (El, & and (gfl 



yields 



lim Jf e [ ,l) : 41^-^)<M/2 =0 a.s. (42) 



n 



Moreover, since Pr |/j (<5 G [0, 1) : c ""^ x ") - c ™(^ x ") < < l} = 1, for all n G Z+ using domi- 

nated convergence theorem, (02) implies that 

C^X") C(<5,X") 



lim Ep x 



M U G [0, 1) 



< M/2 



0, 



which in turn implies that (cf. sufficient condition for channel simulation) Z is an approximating source for 
W, given X. 

2) H(p 2 ) < H(q 2 ) or H( Pl ) < g( gl ): Let M= min{(ff(g 2 ) - H(p 2 ))+ , (H( gi ) - H( Pl ))+} G R+ where 
denotes max{x, 0}, for any x£l. Further, fix some e G (0, 1/4). Recalling the law of total expectation, 
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we have 



i 



„h e [0,l-e):^ + £ - X "'-^ X °'<-^ 
\ n n 2 



^Pr{Q 2 = i}.E 



i=0 



M U € [0, 1 - e) 



c z J x (5 + e,X n ) c%{6,X. n ) 



for any n E Z + . Further, recalling (l39l ) and (l40b . we have 



lim inf E 

n— >oo 



/x (5 E [0, 1 - e) : 



c$ x (5 + e,X n ) <%{8,X« 



n 



< 



M 



l 



> 0. 



(43) 



(44) 



n n 2 

Plugging (l44l into (l43l (since /? > 0), we conclude that (cf. the necessary condition for source simulation) 
X is not an approximating source for Y. 
3) H(pi) = H(qi), H(p2) = H(q2): Although it is possible to simulate Y using X, the results in this paper 
are inconclusive for this case. 
Example 4: Let X n = X (resp. y n ) be some arbitrary set with < oo (resp. \y n \ = y/n), for all n G Z + . For 
any n £ Z + , define X n (resp. Y n ) as the uniform random variable over X n (resp. y n ). Recalling ©, we have 



lim cl(5) = ^logn, lim c%{5) = \og\X\, 

n—>oo Z n—>oo 



(45) 



for all 5 G [0, 1). Recalling the necessary condition for the source simulation, (1431) immediately implies that 
{X n }^ =1 is not an approximating source for {Y n }^ =1 . However, if X n and Y n are independent for all n £ Z + , in 
other words if we let Px n ,K n = Px n PY n , then we have 

1 



p - liminf <! - log ^ : 



n l0g 'PyJ)],) 



> 0. 



(46) 



Recalling ([8]), (l46l ) implies that the necessary condition of [22] is satisfied. 

III. Source Simulation 

In this section, we deal with the problem of approximating a general target source with a general coin source. 
Although this problem is a special case of the channel approximation with a deterministic input to the channel, 
we include it as a separate section for the following reasons: First, the results of this section constitute the 
core of the sufficiency and necessity proofs of channel simulation problem; to be more precise, Proposition 13.11 
(resp. Theorem 13.3b plays a fundamental role in the proof of sufficient (resp. necessary) condition of the channel 
simulation problem. Moreover, for this special case, we prove a stronger necessary condition compared to the single 
source counterpart of the channel simulation necessary condition. Last reason is the problem's particular practical 
importance. 

Throughout this section, let X = {X n }^ =1 and Y = {Y n }^ =1 be two general sources with X n and y n being 
countable sets for all n E Z + . 
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A. Source Simulation-Sufficient Condition 

First, we begin with the following proposition, which constitutes the core of the achievability proofs for both 
source and channel approximation problems. 

Proposition 3.1: Consider any e G (0, 1) and 7 G M + , such that e~ 7 < e. Then, for any n G Z+, 

3 </>„ : ;r n y n , with d(y nj <^(x„)) < 9e + 10^(^(7)), 

where £ n (j) = {8 G [0, 1) : t*(<J) - c v n {8) < 7}. 

Proof: Consider any e G (0, 1) and 7 G M + , such that e~ 7 < e and consider any n G Z + . Define 
«4"(7) = [0, l)\£ ra (7). Next, fix some S n (X) and S n (Y) as given in Definition 12.41 and define the following 
indexes: 

00 

h = M{j G Z+ : P *^) < e i' ( 47 ) 

i=i+i 
00 

Note that both i\ < 00 and 12 < 00. 

Define rj x = Yli^i +1 Px n {xi) (resp. rj y = Yli^=i +1 Py„ (Vi))- Using these, define the following partitions of 
[0,1] 

A* = {0 = ^ < 8f < . . . < 81 = 1 - Tfc}, s.t. Vi, A? = <ff - <f-i = PxM), (49) 
A» = {0 = d»<<S?<...<<$V=l- s.t. Vi, = 5* - 8U = PyM- (50) 
Next, we define the following mapping: 

<t>'n ■ {xi}%i -> {Vi}SLi, s.t. tf>' n (xi)= yj , ifC-lG[^-i^),V*€{l,...,ii}, (51) 

for some corresponding j G {1, . . . , 22}- We define j'2 G {1, . . . , ^2} as the only index with ^(x^J = y j2 . Note that 
J2 < It can easily be verified that (p' n is a well-defined mapping. Now, using (l5TT >. we define (f) n : X n ^ y n , 
such that 

* w ={T' (52) 

As a shorthand, let Y n = <j) n (X n ) denote the output of the mapping. 

After defining the mapping, we analyze the variational distance between Y n and Y n . First of all, using (l52l ). we 
have 

32 00 

d(Y n ,Y n )<Y,\PY n (yi)-P^)\ + \PYM-PY n ^)\+ E p y^- ( 53 ) 

i=l i=32+l 

Next, we need the following sets of indices: 

l={i G{l,...,j 2 } : Py n ( yi )^0}, 
J={iG {!,... ,i 2 } : Pr((^)- 1 ( 2/J ))=0}. 
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Further, we define the following subsets of £"(7): 

Vi € {1, . • • ,j 2 ~ 1}, £i = (<f , <5fe) n £ n ( 7 ), with 5'f G [^_ l5 ^) for some fc G Z+, (54) 
£ j2 = (5l,l-e)n£ n ( 7 ). (55) 
Vi 6 {1, . . . ,i 2 - 1}, £ = n «f n (7), (56) 

4 = (^-i,i-e)nr»( 7 ). (5V) 

Remark 3.1: Note that, in some sense, the aforementioned sets can be thought of as the "bad sets", which will 
contribute to the variational distance. Hence, we aim to relate the resulting variational distance between Y n and Y n 
created by (l52l ) to the measure of these sets. Observe that Vi, j G {1, . . . , 12}, £{ n £j = 0, if % 7^ j, i.e. {Si} 1 ^ is 
a disjoint collection of subsets belonging to £"(7). 

In order to upper-bound (I53T ). we first prove the following lemmata: 

Lemma 3.1: For any i G {1, . . . , 32 — 1}, if 6f G <^f) for some k G Z + , then 

6%-%<ePy n ( yi+1 ) + n(£i). (58) 

Consider any i G {1, . . . , 32 — 1} an d suppose 3 5 G (5f , <5jp, such that (5 ^ £j, i.e. 5 G A n (j), where £j is as 
defined in (|54l . Then, recalling (0]) and the ordering provided by S n (Y), we have 

^ )=1 ° E OT' (59) 

Using definition of ^."(7), we have 

PxM < e-^Pv n (yi+i) < ePr n (yi+i), (61) 
Further, note that Px n { x k) > — <5f- Combining this with d6"TT ) yields, 

^-^<eJV„(2/ i+ l)- (62) 
If ^ <5 G (<5f,^f)> such that (5 ^ £j, then we have (<5f,<^) = £j, which readily implies 

= At(£). (63) 

Combining (l62t and (l63l yields 



<S£-#<eiY B (lfc+i) + M&), 



which was to be shown. 

Lemma 3.2: For any i G X such that i ^ 12 



Py n ( yi ) = n(Si). (64) 
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Further, if %i G X, then 

iV B (ViJ<2e + M(4)- (65) 

Proof: 
Consider any iGl 

First, suppose and observe that by recalling (f5TT >. i Gl implies that 



(66) 

for some G Z + . Next, suppose that 35 G (<5f_ 1; <5f), such that 5 ^ £j, i.e. 5 G ^."(7). Then, recalling (0]) we have 

tS^log—L- (67) 
c ^(5)=log-^— . (68) 

PvAyi) 

Hence, (l67l) and (l68l) implies that we have 

5% - 8U = P Xn (x k ) < e^P Yn ( yi ) < £ $ - Ci) < - Ci, 
which yields a contradiction with d66l ), therefore we conclude that 

(Ci,^) = £- (69) 



( f69b directly implies that Sf — b\_ x = PyiVi) = n(£i), which proves (l64l ). 
Next, suppose 12 G X, which directly implies 

< ( 7 °) 
^(w,) = (^-l + e) + (l-e-^_i) > 

<e + (l-e-5f 2 _ 1 ), (71) 

where ([7T]) follows from (l48l) . 

Next, suppose 35 G (5f 2 _ l5 1 — e), such that 5 ^ £{ 2 , where £i 2 is as defined in (l57l) . Then, recalling (01) and (l70l) 
we have 

log 1 , (72) 

^(5)= log— . (73) 

Hence, d72l) and (1731) implies that we have 

Px„ (xi, ) < e-^P Yn (y i2 ) < ePy n (y i2 ) < e. (74) 

Moreover, 

P Xn (^ 1 )>(l-e-<5F_ 1 ), (75) 
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which follows from dTUb . Combining d74l ) and d75l ) yields 

(l-e-^^e. (76) 
If J<f G (Si 2 - V 1 - e), such that 6 <£ £ i2 , then we have (cf. §7}) 

l-e-C-i = M4)- (77) 



d76l ) and ( 1771 ) implies that we have 



Plugging (1781 ) into (1711 ) yields 



1 <e + /x(4). (78) 
iV B (l/i a )<2e + /i(4), 



which was to be shown. ■ 
Next, in order to rewrite <i(Y n , in terms of £"(7), we need disjointness relations between elements of {£i}i & x, 
even if this collection may not be necessarily pairwise disjoint. To this end, we need following sets: 

{i € X : 1 < i <j 2 - 1, 3k x G Z+ s.t. Ci G 3fc 2 € Z+ s.t. # G (^ 2 _!,C)} (79) 

{* € X : 1 < * < h - 1, 3h G Z+ s.t. G (d^, <5&), 3£; 2 G Z+ s.t. 6f = 5 x k J (80) 

{i G X : 1 < i < j 2 - 1, 3h G Z+ s.t. 5f_ x = 3A: 2 G Z+ s.t. 5* G (^ 2 _ l5 (81) 

{« G X : 1 < * < j 2 - 1, 3fci G Z+ s.t. 5f_ x = 3k 2 G Z+ s.t. <J* = 5fJ (82) 

Note that {Ii}f =l forms a partition of X\{j 2 }. 
First, suppose i G {1, . . . , j 2 — 1}- 
Now, if i G X\j2, then using (I5TI ). we have 

' (^-Ci)+i^ 2 -^)> if^Xo,o, 

|JV„(w) I < < 





A 


Xo,o 




Xd,i 


A 






A 


Xi,o 




Xi,i 


A 





if*GX 0)1 

(^-<5?), if i€ Ji, , 

0, if*€Ji,i. 



(83) 



We have 



3 

^ |JV„(lfc) - Py„(l«)| = E E l^-Cw) - (84) 

jglo.o «€Xo,i 

+ £ (eJVn(lfc+i) + M&)), (85) 

where (|84l ) follows from d83l , d85l ) follows from ( T58T ), d86l ) follows from definition of S n (Y) and disjointness of 
XcoiXo.ijXi^. Observe that 1 G Xi o UZii from the definition, so in (|86l ) we never refer to fi(£o), which is not 
defined. 
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In order to bound ( 1861 ), we need following result. 

Lemma 3.3: {Si} iEloo , {£i-i}iex ,o> {£i-i}iei .i and {^}ieJi. are disjoint collection of subsets of £"(7). 
Proof: 

We prove following two claims, which are sufficient to conclude the result. 

Claim 3.1: For any i,j G X, such that i,j ^ j'2 and i ^ j, if 3A; G Z + with 5f G (S^._ l ,S^) and 3/ G Z + with 
b) G (5f_ l9 <5f ), then Si n £, = 0. 
Proof: 

Consider any i,j G X, such that i, j 7^ ji and i 7^ j, with £V G 5%) and <5j G (Sf_ l ,5f), for some fc, Z G Z + . 

W.l.o.g. suppose i < j, which implies that k < I. If k = I, we have [tfjLj,^] C (£fc_i>£f)> which contradicts 
j G X, hence we should have k < I, which immediately implies that {8f, <5|) n ($j,5f) = 0, which was to be 
shown. ■ 

Claim 3.2: For any i,j G X, such that i, j / ji and i 7^ j, if 3k G Z + with 5f_ 1 G (^_i,^f) and 3/ G Z + with 
<^_i e (5f_ v 5f), then £_! n = 0. 
Proof: 

Consider any i,j G X, such that i,j ^ 22 and i / j, with <5f_ x G {5f_ 1 ,S^,) and G ((5f_ 1; (5f), for some 
k, I G Z + . W.l.o.g. suppose j < j, which immediately implies that k < I. Suppose, for contradiction, k = I. 
Since i < j, we have 6f < <5j_i> which implies that [5f_ l5 <5f] C (^k-i^k)' which in turn implies that i G X, 
which contradicts the assumption i G X, hence we have k < I. Using this, we have 5| < 6f_ ls which implies that 
(SV_ 1: Sf) n (<5|_i, <5f ) = 0, which implies the result by recalling (l54i ■ 

Now, observe that Claim |3~T1 implies that {Si}i£i , {£i}i&z 10 are disjoint collection of sets, while Claim [3721 
implies that {£j_i}j £ x , {£j_i}j e x 1 are disjoint collection of sets, which was to be shown. ■ 

Using the result of Lemma 13.31 in (l86l) we have 

j'2-1 

^ |JV B (l/i) - P ?B (y<)| < 2e iV B (%) + 4/i(f n ( 7 ))- (87) 
Next, suppose i G {1, . . . , j'2 — 1} and i G X. We have (note that 1 G X as a result of (1571 )) 

^ |iV B (|/i)-Py>i)|= (88) 

Ki<j 2 , S.t. ieX Kt-Q'a, S.t. igX 

< 2 (89) 

Ki<j 2 , S.t. ieJ 

< K£ n (i)), (90) 

where ([881 ) follows from definition of X, (l89l) follows from Lemma I3T21 and (l90l) follows since {<?j}*? =1 is a disjoint 
collection of subsets of £"(7). 

Now, we complete the proof. First, suppose that ji = i%. Using (l52l ). we have 



|Py„(y i2 )-Py„(^)l<|Pr{(^)- 1 (y l2 )}-Py„(y« 2 )l+ £ Px B (*i). (91) 



i=ii+l 
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Further, using the result of Lemma [3TT1 and \6f — Sf \ < e (cf. definition of i% and %i) we have, 

\PyM ~ P^C^J" 1 ^)}! < zPrM + e + //(f n (7))- (92) 
Moreover, the definition of i\ immediately implies that 

oo 

J2 Px n (X n )<e. (93) 
t=ii+l 

Plugging (|93) and (|92]> into (HD yields 

\PrM - P%{yi,)\ < ePy n (yi 2 ) + ^ + K£ n (i))- (94) 

Further, recalling definition of we have 

oo 

PrAyi)<£- (95) 

i=i 2 +l 

Plugging ©, dH and <|95) into 63) yields 

rf(K„,K„) <9e + 7^(£: n (7)), (96) 

Next, suppose that j'2 < «2- First of all, note that 5^ = 5^ is not possible for any k G {1, . . . , ij — 1}, because if 
it is the case, then <j)' n {xk) = Uj 2 +i, which contradicts with the definition of ji- Further, observe that 4>n{, x ix) = Vj 2 ' 
which directly follows from (I5TI ). Using these observations, we have 



for some k < i\ with $% i G [^fc-u^fc)- 

If 3 5 G (<5j 2 , 1 — e), such that 5 ^ £j 2 , then using similar arguments as in the proof of Lemma |3~T1 it can be 
shown that 

l-e-5 y j2 <eP Yn (y j2+ i)- (98) 

If $6 G (<5| 2 , 1 - e), such that 5 £ £ ja , then ($V , 1 -e) C £ h , which implies that 1 - e - 5 v h < fi(£ h ) < n{£ n {i)). 
Combining this with d98l ) yields, 



i-e-q a < eP Yn {y 32+ i) + K£ n d))- (99) 

Further, 6f — 1 + e < e. Combining this with (|99l and noting that -Py„(2/j 2 +i) ^ PY n (yj 2 ) (cf- definition of <S n (y)) 
yields, 

5f t - S» h < e + ePrAVh) + ti^W). (100) 
Using (II 001 ) in d9Tb and recalling the result of Lemma 13.11 yields 

|Pr„(% 2 ) " Py n (yh)\ < ZzPyM + e + 2/u(^( 7 )). (101) 
Next, using (|52l ) and recalling the assumption of j% < %2, we have 

liVnG/iJ " Py n {y i2 )\ < PyM + (1 " < 3e + ^ n (7)), (102) 
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where second inequality follows from Lemma [372] and the definition of i\. 
Further, 

oo i 2 oo 

E p ySvi)= E p yM+ E ^-(f*) 

i=j2+l «=J2+1 i=«2+l 

«2 

< E p ^(^) + e ' ( 103 ) 

i=j2+l 

where dl03l ) follows from d95l ). 

Now, suppose j2 = ?2 — 1- Then, (11031 ) reduces to 

oo 

E PrM <^(^ 2 ) + e<3e + ^ n (7)), (104) 
i=j2+l 

where second inequality follows using Lemma 13.21 Hence, plugging ((871 ), (|90l , dlOll ), (11021 ) and dl04| ) into d53l ) 
yields, 

d(^n,^n) <9e + 9^(£: n (7)). (105) 
Finally, suppose j'2 < «2 — 1, using Lemma [3721 in d 103b . we have 

oo i 2 ~ 1 

E *V„(vi)< E M£) + 3e + M£ n (7)), 

< 3e + 2/i(£ n ( 7 )), (106) 

where (11061 ) follows from the fact that {<£'i}*L 1 is a disjoint collection of subsets of £"(7). Hence, plugging d87l ), 
(HB, flOB, (HSU and CE06]> into 63) yields, 

<r n ,Y n ) <9e + 10/i(£ n ( 7 )). (107) 

Lastly, combining d96l ), (11051 ) and (11071 ) yields the sought after result. ■ 
Theorem 3.2: If 

^-lminf{c2 (<$)-<*(<$)} = 00 (108) 

n — >oo 

then X is an approximating source for Y. 
Proof: 

First, observe that (11081 ) is equivalent to 

V7 G M,Ve G M + , 3iV(e,7) G Z+ s.t. Vn > JV(e, 7), y. {5 G [0,1) : c x n {5) - c y n {5) < 7) < e. (109) 

Next, consider any e G (0, 1) and 7 G M + , such that e~ 7 < e and fix some n > N(e, 7). (11091 ) immediately implies 
that (recall definition of £"(7) in Proposition 13.11 ) 

M£ n (7))<e- (HO) 

Proposition 13. 1 1 immediately implies that 

3cf> n : X n - y n , with d(y n ,y n ) < 19e, (111) 
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where Y n = <p n (Y n ). Next, we define 

Vn € Z+ <= inf{d(y n ,y n ) : 3<£ n : Af n -»■ ^ n }- (112) 
(II 1 II ) implies that Vn > 2V(e, 7), cZ* < 19e. Since e G (0, 1) is arbitrary, this implies that we have 

limsupci* = 0. (113) 

n^oo 

Lastly, for each n G Z + , let choose <j> n : X n — ► 3^ n , such that ci(Y n , Y n ) < 2ci* and form the sequence of mappings 
{4> n : X n — > 3-nK£Li- F° r this sequence of mappings, we have 

limsupd(Y n , Y n ) < 21imsupd* = 0, (114) 

n^oo n— >oo 

where equality follows from (11 13b . Using the definition of variational distance and (11141 ). we have 

< lim inf d(Y n ,Y n ) < limsup d{Y n , Y n ) < 0, 

n >oo n — >co 

which implies that lim n ^ 00 d(Y n , Y n ) = 0, which was to be shown. ■ 

B. Source Simulation-Necessary Condition 

Theorem 3.3: If X is an approximating source for Y, then 

inf hm inf inf {<£($ + €)-#($)}><), (115) 

0<e<l n-*oo 0<5<l-e 

where c^(<5) and c«(5) are as defined in (0]). 
Proof: 

Let {4> n : X n -> ^nj^i be a sequence of mappings such that lim n ^oo d(Y n , 4> n (X n )) = and Y n = <f> n (X n ). 

We will prove the theorem in two main steps. In the first step, we prove another condition, which essentially 
states that if X is an approximating source for Y, then asymptotically, the cumulative distribution of its entropy 
spectrum is greater than that of Y. In the second step, we prove that the aforementioned condition implies (11 151 ). 

Before stating the proof, we need the following definition: 

Definition 3.1: Let U, V be real valued random variables. The Levy distance between them, denoted as L(U, V), 
is defined as 

L(U,V) = inf {/M G R + : Vx G R,Pt{U < x - u} - £i < Pr{V < x} < Pr{[/ < x + u} + u} . (116) 



We continue with the following lemma 
Lemma 3.4: For Y n ,Y n G y n , 



If li.n d[Y„.Y n ) -U. then liin I [ log ) log 1 ) =0. (117) 

^y„l r n) Py n {Y n ) I 



n— »oo V / n— >oo 



Proof: 
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Proof readily follows from the same arguments used in Theorem 2.1.3 of [12], of which RHS is 
lim n ^oo L (~ log p Y ^(y ) > h 1°S P . ) ^ = 0' anc ^ tne particular proof does not depend on the existence of 1/n 
factors. ■ 

Next, we prove the following lemma, which is the entropy spectrum counterpart for the well-known fact [24] 
'any deterministic mapping of a random variable cannot increase its entropy.' 

Lemma 3.5: For any re G Z + , let X n be a random variable taking values in X n , where X n is countable set and 
Y n = (fi n {X n ), where <fi n : X n — » y n is any deterministic mapping and y n is countable set. Then, we have 

Vce R - p >-- ( log ^k) * c ) - Px - ( log ^ *= c ) • (118) 

Proof: Define the following sets: 

T*(c) = {xG X n :P Xn (x) <e~ c }, 
TX(c) = {v G : PySv) < e" C } • 

If we can show that 

Pr {T„ x (c)} - Pr {if (c)} > 0, (119) 

for arbitrary choices of c and re, then this will conclude the proof. Now, observe that since Y n is a deterministic 
function of X n , we have 

Vy g ^n,Py n (y) = Px„ faHv)) ■ (120) 
Using (11201) and recalling the definition of T„ , we have 

PrfrJ (c)} = Px„ fx G * n : x G 0~ X fee))) . (121) 



Now, if T%(c) = 0, then (fTT9l holds, hence we are done. Suppose this is not the case. Then, we choose any 
x G (fi^ 1 (t^(c)\. We know that 4> n {x) G T^(c), hence Px n {x) < e~ c (otherwise (fi ri {x) cannot be an element 
of T% by recalling the definition of this set and dl20| )), which implies x G T^(c). Since x G (fi~ l (t^ '(c) J is 
arbitrary, we conclude that 

■ lf T*{cj)ci*(c). (122) 



(TT2Tb and (fT22l immediately implies that Pr{T,f (c)} > Pr{^~ 1 (T^(c))} = Pr{T*"(c)}, hence (TTT91) holds. ■ 
Lemma 3.6: If X is an approximating source for Y, then 

^ *ag si { ( los ft^vy < c + ") " Px - ( log *d*o < c ) } - {m> 



Proof: 
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Suppose X approximates Y. Then, using dl 17K we have 

inf lim inf inf P Y (log < c + u! ) + u! - P-y ( log ^— < c | > 0. (124) 



Now, observe that we have 



v/'' 6 P+. P Y „ ( log ^ 1 - <c+A+fi'<PY n (log p^Y) < c + V j + 2//'. ( 125 ) 



1 \ „ /. 1 



Further, 

Vc e M - Pfc t log £ c ) - Pfc t log w * 1 - Px - l log <c] - im> 

where second inequality follows recalling (111 8b - 

Using (11251 ) and (11261 ) in (11241 ) (by defining n = 2//) yields 

inf liminf inf I P Y „ (log * r <c + m) +^-P Xn (log ) r < c J 1 > 0, 



which further implies 

liminf liminf inf { P Yn (log - \ < c + fj, J - P x „ (log 75 — 777^- < c 

lim inf lim inf inf { P Yn (log 1 < c + /x j + y. - P Xn ( log * < c J 1 > 0. (127) 

M ->0+ n->oc ceR [ V -fV„( y n) J \ PxA X n) ) J 

Now, observe that P Yn (log Py jy ^ < c + fij is non-increasing with decreasing fi, therefore (11271 ) implies that 
inf liminf inf { iV„ (log - ^7 . < c + ^ ] - Px„ (log 7; — 7— r < c 
liminf liminf inf \P Yn (log 1 < c + /* J - P Xn (log * . < c J } > 0, 

M ^ + n^oo ceR [ y Py n {Y n ) ) \ PxA X n) J) 

which was to be shown. ■ 
Now we finished the first step of the proof. What remains is to prove the following lemma, which is the second 

step of the proof. 
Lemma 3.7: If 

inf liminf inf { P Yn (log ) <c + fJ \- Px (\ og 1 < cH > 0, (128) 

M £R+ n^oo c6 R [ \ P Yn (Y n ) J "V PxA X n) J J 



then we have 



inf liminf inf cf(<5 + e) - &,{5) > 0. 

eG(0,l) n^oo S£[0,l-e) 



Proof: 

First observe that using (11281 ). we have 

V// G M+, liminf inf \ P Yn (log ) <c + fi)-P Xn (log ) y < c ) } > 0. (129) 

n->oo cG R [ y PY n {Y n ) J V PxA X n) 
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Now consider any e G (0, 1). Next, fix an arbitrary p G M + and a sufficiently large n, such that we have 

Ma { Px » ( log ^kj - * (1 - 5) - ") - ( log ft^k) - 15(1 " {) ) } - " e - <130) 

Consider any 5 G (0, 1] and observe that we have (cf. definition of c%,{8)) 

^( log ^k)- c " (1 " 5) )- 5 - (131) 

Moreover, using (11301 ) and (11311 ). we have 

Px n (log p ^ | > <(1 - ,5) _ ^ > <5 - €. (132) 
Recalling the definition of c^(<5), d 1 32b immediately implies 

c x n (l - 5 + e) > cKl - 5) - fi, (133) 
for any e G (0, 1). Since \i G M + is arbitrary, (11331 ) implies that we have 

liminf inf {c*(l - <5 + e) - t*(l - 5)} > 0. (134) 

n->oo 5g( e ,l] 



(fl34l yields 



liminf inf {c x n {5 + e) - c y n (5)} > 0. (135) 

ra— >oo 5g[o,l- e ) 



Since e G (0, 1) is arbitrary, (11351 ) yields 

inf liminf inf + e) - 4(5)} > 0. 

ee(0,l) n->oo 5G[0,l-e) 

which was to be shown. ■ 

Combining Lemma 13.61 and 13.71 we conclude that (11 15b follows. ■ 

C. Source Simulation-Comparison of the Necessary Condition to Its State-of-the-Art Counterpart 

In this section, we demonstrate that the necessary condition of Theorem [33] is strictly stronger than the necessary 
condition of [22] (cf. ([8])), which is valid for only finite alphabets. First, we prove the following theorem. 
Theorem 3.4: If (11151 ) holds, then 

Vn, 3 P X v , with marginals P Xn , Py n , s.t. p - lim inf j - log - \ - - log - ] 1 > 0, (136) 

n^oo [ n P Xii (X n ) n P Yll {Y n ) ) 

where the probability measure is P Xni y n - 
Proof: 

First, we prove the following lemma. 
Lemma 3.8: If ( 11151 ) holds, then 

Vn G Z + , 3{P Xn ,Y n } n >i, with marginals P Xn ,Py n , s.t. p - liminf { log - — - log - 1 1 > 0. (137) 

n-»oo [ P Xn {X n ) P Yn {Y n ) ) 



Proof: 
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dl 151 ) is equivalent to 

VeG (0,1), V 7 G M+,3iV(e,7) G Z+ s.t. Vn > iV(e, 7 ), V5 G [0,1-e), c£(5 + e) - c^(<5) > -7. (138) 
(11381 ) implies that 

Ve G (0,1), V7 G M + ,3iV(e,7) G Z+, s.t. Vn > iV(e, 7 ), /x(<5 G [0, 1 — e) : c x n {5 + e) - c y n {5) < -7) = 0. (139) 
Consider any e G (0, 1) and 7 G R + and fix some n > N(e, 7). Next, define Q, ~ t/[0, 1] and 

Let <S n (X), A 21 and c^(5) (resp. 5 n (y), A y and c«(5)) denote the quantities defined in ©, ([3]) and dU), respectively, 
for X n (resp. 3-n)- Using the Q, Q pair, define the following joint distribution for X n , Y n 

Px n ,Y n (^Vi) = Pr jc*(fi) = log 4(12) = log ^ - I , (140) 

for any G {1, . . . , |5 n (X)|} x {1, . . . , |<S„(Y)|}. Note that as a direct consequence of (11401 ). the marginals of 
Px n ,Y„ are Px n and Py n . Further, we have 

- { lo ^ P^k) " bg PrJXn) K " 7 } = Pr {'--- ; - C " (0) < " 7 1 ' (141) 

< Pr {u; G [0, 1 - e) : c x n (u + e) - t*(u) < -7} + e, 

< M (J G [0, 1 - e) : c£(eJ + e) - < -7) + e, (142) 

< e, (143) 

where (11411 ) follows from (1140b . (11421 ) follows from the definition of and O, and (11431 ) follows from (I139I ). 
Since e G (0, 1), 7 G R + and n > N(e, r y) is arbitrary, (11431 ) implies that 

V 7 G R+ lim P x „, r „ flog - ) - log - 1 < -7) = 0. (144) 

Recalling definition of limit infimum in probability, (11441 ) implies that for this particular construction of Px n ,Y n , 
we have 



p-liminf (log - ) -log ) v , \ >0, 



where probability measure is Px n ,Y n > which was to be shown. ■ 
Next, we conclude the proof. Lemma 13781 directly implies that if (II 15b holds, then 3{Px n ,Y n }n>l with marginals 
Px n and Py n , such that 

Ve, 7 G R + , 3N(e, 7) G Z+ s.t. Vn > JV(e, i),P Xn ,Y n (- log D * , - - log - \. . < -- J < e. (145) 



°P lB (X n ) n P Yn (Yn) n 
(11451 ) immediately implies that for this sequence of joint distributions, we have 

p - liminf 1 - log ) . - - log * r 1 > 0, 

n-*oo [ n Py n (A n ) n Py„(y n )J 

where probability measure is Px„,Y n - Hence we are done. 
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Remark 3.2: If we take X n (resp. y n ) as a finite set for all n G Z + , then (11361 ) implies the necessary condition 
of [22], hence Theorem 13.41 implies that our necessary condition is at least as good as the one in [22]. Moreover, 
recalling Example 4 stated in Section ITl-C I the converse is not true, in other words ([8]) does not imply our necessary 
condition. Therefore, we conclude that our necessary condition for the source simulation problem is strictly stronger 
than its state-of-the-art counterpart. 

IV. Channel Simulation 

Throughout this section, X = {X n }^ =1 and Z = {Z n }^ =1 denote two general sources, where for all n, X n is 
a random variable taking values in X n , such that X n is countable set (resp. Z n is a random variable taking values 
in Z n , such that Z n is countable set) with an arbitrary coupling Pz n \x n - Further, ~Wy\x = {^y„|x„(^ / n|^n)}^ 1 
denotes a general channel, where for all n G Z + , Wy n \x n denotes a conditional pmf over y n x X n , with y n being 
countable set. 

Theorem 4.1: If 

V 7 G R, lim E Pxn \n(6 G [0, 1) : c z J x (5, X n ) - c™(<5, X n ) < 7 )1 = 0, (146) 

n— >oo L . 

then Z is an approximating source for W, given X. 



Proof: 

First, observe that (11461 ) is equivalent to 

VeGM + , 7 GM, 3iV(e, 7 )GZ+, s.t. Vn > AT(e )7 ),E Pxn [m(* € [0, 1) : < |a: (£,X n ) - c£ < 7 )] < e. 

(147) 

Next, consider any e G (0, 1), 7 G M + , such that e~ 7 < e and fix some n > N(e, 7 ). Now, observe that for any 
x G A? n , both Z n and Y n are general sources with distributions Pz n \x n {'\ x ) an( ^ ^V„|x„('l x )- F° r anv x £ X n (by 
defining £ n ( 7 , x) = {<5 G [0, 1) : d^ z (5,x) — c™{8,x) < 7 }) Proposition 13. II guarantees that we have 

3<f> x n : Z n -► 3> n , with 4 < 9e + 10/x(£ n ( 7 , x)), 

where 



Vx G * n , d x =Y. \ W Y n \xM*) - Pz n \x n {{4> x n r\y)\x)\ ■ 
Using 4>n' we define the following mapping 



We have 



ip n : X n x Z n -> s.t. V(x, z) G A? n x Z n , ^(x, z) = ^(z). 

d(X n Y n , X n (p n (X n , Z n )) = Ep Xn [dx„] , 

< 19e, 



(148) 
(149) 

(150) 

(151) 
(152) 
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where dl51| ) follows from dl49l ), (11501 ) and recalling definition of variational distance and dl52| ) follows from (11471 ) 
and (fl48l 

Using arguments similar to those of the proof of Theorem 13.21 one can conclude the proof. 



Theorem 4.2: If Z is an approximating source for W, given X, then 

Ve G (0, 1), V 7 G R+ lim Ep„ L(5 G [0, 1 - e) : c z J x (5 + e,X n ) - t%(6,X n ) < -7) 



0. (153) 



Proof: Let {(^ n : x Z n — > 3^}^! be a sequence of mappings with 

lim d(X n Y n , X n ip n (X n , Z n )) = 0. (154) 

n^oo 

Let iV G Z + be such that Vn > iV, Ep [dx n ] < L where dx„ is as defined in (I149I ). For the sake of notational 
convenience, we define 

£(e, 7, Xn) = M (<* G [0, 1 - e) : + e, x n ) - c™(5) < -7) , (155) 

for any e G (0, 1), 7 G R + and x n G X n . Next, we consider any n > N and define the following set: 

S n = {x n G X n : d x „ > (E Pxn [d Xn }) 1/2 }. (156) 

Using Markov's inequality, (11561 ) implies that 

Pr{X n GcS n }<(E Px Jd x J) 1 / 2 . (157) 

Moreover, define x = {i n }™ =1 , where Vn G Z + , x n G X n and for all n > N, x n G and /i(e, 7, x n ) > 
5 su Pi„eS c A( e i7i x n)- Observe that (11571 ) guarantees the existence of such a sequence, since 5^ 7^ for all 
n > iV. By the definition of x, we have lim^^co d^ n = 0, hence the necessary condition of source simulation, i.e. 
(II 15b . implies that 

lim /i(e,7,x n ) = 0, (158) 

n— >oo 

for any e G (0, 1) and 7 G M + . 

Consider any n > N, e G (0, 1) and 7 G M + . Using the law of total expectation, we have 

E Pxn [/i(e, 7, X n )] = E[/2(e, 7, X n ) I X n G 5 n ] • Pr {X n G 5 n } + E[/x(e, 7, X n ) \ X n G S c n ] • Pr {X n G S c n ] , 

< (Ep x Jd Xn ]) 1/2 +E[/i(e,7,X n ) |X n G (159) 

< (E Px?i [d x J) 1/2 + 2/}(e, 7, x n ), (160) 
where dl59l ) follows from (11571 ) and d 1 601 > follows from the definition of x. dl60l ) implies that 

limsupEp [/i(e,7,X n )] < lim (E Px [d Xri }) 1/2 + 2 lim p,(e,^f,x n ), 

< 0, (161) 

where (fToTp follows from (fl54l and (fT58T ). Since Ep Xn [/2(e, 7, X n )] > 0, Vn G Z+ and e G (0,1), 7 G M+ are 
arbitrary, (1161b implies (1153b - ■ 
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V. Conclusion 

In this paper, we consider source and channel simulation problems for the general case and prove essentially 
the same necessary and sufficient conditions. The necessary and sufficient conditions for the source (resp. channel) 
simulation problems exploits the knowledge of the whole entropy (resp. conditional entropy) density of the target 
source (resp. general channel) and the coin source. Moreover, our necessary condition for the source simulation 
problem is strictly stronger than its state-of-the-art counter part (cf. [22]) which is valid for only finite alphabets. As 
a future research problem, this kind of approach may also be exploited to solve the general case of the approximation 
theory of output statistics (which is originally formulated in [17] for the special case of fair coin flips as the coin 
source) problem which is still an open problem (cf. [23]). 
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